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Abstract 


This document provides a survey of transport protocols that are 
designed to have a smaller bandwidth and/or delay impact on standard 
TCP than standard TCP itself when they share a bottleneck with it. 
Such protocols could be used for delay-insensitive "background" 
traffic, as they provide what is sometimes called a "less than" (or 
"lower than") best-effort service. 
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1. Introduction 


This document presents a brief survey of proposals to attain a Less- 
than-Best-Effort (LBE) service by means of end-host mechanisms. We 
loosely define an LBE service as a service which results in smaller 
bandwidth and/or delay impact on standard TCP than standard TCP 
itself, when sharing a bottleneck with it. We refer to systems that 
are designed to provide this service as LBE systems. With the 
exception of TCP Vegas, which we present for historical reasons, we 
exclude systems that have been noted to exhibit LBE behavior under 
some circumstances but were not designed for this purpose (e.g., 
RAPID [Kon09]). 


Generally, LBE behavior can be achieved by reacting to queue growth 
earlier than standard TCP would or by changing the congestion- 
avoidance behavior of TCP without utilizing any additional implicit 
feedback. It is therefore assumed that readers are familiar with TCP 
congestion control [RFC5681]. Some mechanisms achieve an LBE 
behavior without modifying transport-protocol standards (e.g., by 
changing the receiver window of standard TCP), whereas others 
leverage network-level mechanisms at the transport layer for LBE 
purposes. According to this classification, solutions have been 
categorized in this document as delay-based transport protocols, non- 
delay-based transport protocols, upper-layer approaches, and network- 


assisted approaches. Some of the schemes in the first two categories 
could be implemented using TCP without changing its header format; 
this would facilitate their deployment in the Internet. The schemes 


in the third category are, by design, supposed to be especially easy 
to deploy because they only describe a way in which existing 
transport protocols are used. Finally, mechanisms in the last 
category require changes to equipment along the path, which can 
greatly complicate their deployment. 
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This document is a product of the Low Extra Delay Background 
Transport (LEDBAT) working group. It aims at putting the congestion 
control algorithm that the working group has specified [Shall] in the 
context of the state of the art in LBE transport. This survey is not 
exhaustive, as this would not be possible or useful; the authors have 
selected key, well-known, or otherwise interesting techniques for 
inclusion at their discretion. There is also a substantial amount of 
work that is related to the LBE concept but does not present a 
solution that can be installed in end-hosts or expected to work over 
the Internet (e.g., there is a Diffserv-based, Lower-Effort service 
[RFC3662], and the IETF Congestion Exposure (CONEX) working group is 
developing a mechanism which can incentivize LEDBAT-like 
applications). Such work is outside the scope of this document. 


2. Delay-Based Transport Protocols 


It is wrong to generally equate "little impact on standard TCP" with 
"small sending rate". Without Explicit Congestion Notification (ECN) 
support, standard TCP will normally increase its congestion window 
(and effective sending rate) until a queue overflows, causing one or 
more packets to be dropped and the effective rate to be reduced. A 
protocol that stops increasing the rate before this event happens 
can, in principle, achieve a better performance than standard TCP. 


TCP Vegas [Bra94] is one of the first protocols that was known to 
have a smaller sending rate than standard TCP when both protocols 
share a bottleneck [Kur00] -- yet, it was designed to achieve more, 
not less, throughput than standard TCP. Indeed, when TCP Vegas is 
the only congestion control algorithm used by flows going through the 
bottleneck, its throughput is greater than the throughput of standard 
TCP. Depending on the bottleneck queue length, TCP Vegas itself can 


be starved by standard TCP flows. This can be remedied to some 
degree by the Random Early Detection (RED) Active Queue Management 
mechanism [RFC2309]. Vegas linearly increases or decreases the 


sending rate, based on the difference between the expected throughput 
and the actual throughput. The estimation is based on RTT 
measurements. 


The congestion-avoidance behavior is the protocol’s most important 
feature in terms of historical relevance as well as relevance in the 
context of this document (it has been shown that other elements of 
the protocol can sometimes play a greater role for its overall 
behavior [Hen00]). In congestion avoidance, once per RTT, TCP Vegas 
calculates the expected throughput as WindowSize / BaseRTT, where 
WindowSize is the current congestion window and BaseRTT is the 
minimum of all measured RTTs. The expected throughput is then 
compared with the actual throughput, measured based on recent 
acknowledgements. If the actual throughput is smaller than the 
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expected throughput minus a threshold called "beta", this is taken as 
a sign of congestion, causing the protocol to linearly decrease its 
rate. If the actual throughput is greater than the expected 
throughput minus a threshold called "alpha" (with alpha < beta), this 
is taken as a sign that the network is underutilized, causing the 
protocol to linearly increase its rate. 


TCP Vegas has been analyzed extensively. One of the most prominent 
properties of TCP Vegas is its fairness between multiple flows of the 
same kind, which does not penalize flows with large propagation 
delays in the same way as standard TCP. While it was not the first 
protocol that uses delay as a congestion indication, its predecessors 
(like CARD [Jai89], Tri-S [Wan91], or DUAL [Wan92]) are not discussed 
here because of the historical "landmark" role that TCP Vegas has 
taken in the literature. 


Delay-—based transport protocols that were designed to be non- 
intrusive include TCP Nice [Ven02] and TCP Low Priority (TCP-LP) 
[Kuz06]. TCP Nice [Ven02] follows the same basic approach as TCP 
Vegas but improves upon it in some aspects. Because of its moderate 
linear-decrease congestion response, TCP Vegas can affect standard 
TCP despite its ability to detect congestion early. TCP Nice removes 
this issue by halving the congestion window (at most once per RTT, 
like standard TCP) instead of linearly reducing it. To avoid being 
too conservative, this is only done if a fixed predefined fraction of 
delay-based incipient congestion signals appears within one RTT. 
Otherwise, TCP Nice falls back to the congestion-avoidance rules of 
TCP Vegas if no packet was lost or standard TCP if a packet was lost. 
One more feature of TCP Nice is its ability to support a congestion 
window of less than one packet, by clocking out single packets over 
more than one RIT. With ns-2 simulations and real-life experiments 
using a Linux implementation, the authors of [Ven02] show that TCP 
Nice achieves its goal of efficiently utilizing spare capacity while 
being non-intrusive to standard TCP. 


Other than TCP Vegas and TCP Nice, TCP-LP [Kuz06] uses only the one- 
way delay (OWD) instead of the RTT as an indicator of incipient 
congestion. This is done to avoid reacting to delay fluctuations 
that are caused by reverse cross-traffic. Using the TCP Timestamps 
option [RFC1323], the OWD is determined as the difference between the 
receiver’s Timestamp value in the ACK and the original Timestamp 
value that the receiver copied into the ACK. While the result of 
this subtraction can only precisely represent the OWD if clocks are 
synchronized, its absolute value is of no concern to TCP-LP, and 
hence clock synchronization is unnecessary. Using a constant 
smoothing parameter, TCP-LP calculates an Exponentially Weighted 
Moving Average (EWMA) of the measured OWD and checks whether the 
result exceeds a threshold within the range of the minimum and 
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maximum OWD that was seen during the connection’s lifetime; if it 
does, this condition is interpreted as an "early congestion 
indication". The minimum and maximum OWD values are initialized 
during the slow-start phase. 


Regarding its reaction to an early congestion indication, TCP-LP 
tries to strike a middle ground between the overly conservative 
choice of _immediately_ setting the congestion window to one packet, 
and the presumably too aggressive choice of simply halving the 
congestion window like standard TCP; TCP-LP tries to delay the former 
action by an additional RTT, to see if there is persistent congestion 
or not. It does so by halving the window at first in response to an 
early congestion indication, then initializing an "inference time-out 
timer" and maintaining the current congestion window until this timer 
fires. If another early congestion indication appeared during this 
"inference phase", the window is then set to 1; otherwise, the window 
is maintained and TCP-LP continues to increase it in the standard 
Additive-Increase fashion. This method ensures that it takes at 
least two RTTs for a TCP-LP flow to decrease its window to 1, and 
that, like standard TCP, TCP-LP reacts to congestion at most once per 
RTT. 


Using a simple analytical model, the authors of TCP-LP [Kuz06] 
illustrate the feasibility of a delay-based LBE transport by showing 
that, due to the non-linear relationship between throughput and RIT, 
it is possible to avoid interfering with standard TCP traffic even 
when the flows under consideration have a larger RTT than standard 
TCP flows. With ns-2 simulations and real-life experiments using a 
Linux implementation, the authors of [Kuz06] show that TCP-LP is 
largely non-intrusive to TCP traffic while at the same time enabling 
it to utilize a large portion of the excess network bandwidth, which 
is fairly shared among competing TCP-LP flows. They also show that 
using their protocol for bulk data transfers greatly reduces file 
transfer times of competing best-effort web traffic. 


Sync-TCP [Wei05] follows a similar approach as TCP-LP, by adapting 
its reaction to congestion according to changes in the OWD. By 
comparing the estimated (average) forward queuing delay to the 
maximum observed delay, Sync-TCP adapts the Additive-Increase 
Multiplicative-Decrease (AIMD) parameters depending on the trend 
followed by the average delay over an observation window. Even 
though the authors of [Wei05] did not explicitly consider its use as 
an LBE protocol, Sync-TCP was designed to react early to incipient 
congestion, while grabbing available bandwidth more aggressively than 
a standard TCP in congestion-avoidance mode. 
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Delay-—based congestion control is also the basis of proposals that 
aim at adapting TCP’s congestion avoidance to very high-speed 
networks. Some of these proposals, like Compound TCP [Tan06] [Sri08] 
and TCP Illinois [Liu08], are hybrid loss- and delay-based 
mechanisms, whereas others (e.g., NewVegas [Dev03], FAST TCP [Wei06], 
or CODE TCP [Chal0]) are variants of Vegas based primarily on delays. 


2.1. Accuracy of Delay-Based Congestion Predictors 


The accuracy of delay-based congestion predictors has been the 
subject of a good deal of research, see, e.g., [Bia03], [Mar03], 
[Pra04], [Rew06], [McC08]. The main result of most of these studies 
is that delays (or, more precisely, round-trip times) are, in 
general, weakly correlated with congestion. There are several 
factors that may induce such a poor correlation: 


o Bottleneck buffer size: in principle, a delay-based mechanism 
could be made "more than TCP friendly" _if_ buffers are "large 
enough", so that RTT fluctuations and/or deviations from the 
minimum RTT can be detected by the end-host with reasonable 
accuracy. Otherwise, it may be hard to distinguish real delay 
variations from measurement noise. 


o RIT measurement issues: in principle, RIT samples may suffer from 
poor resolution, due to timers which are too coarse-grained with 
respect to the scale of delay fluctuations. Also, a flow may 
obtain a very noisy estimate of RTTs due to undersampling, under 
some circumstances (e.g., the flow rate is much lower than the 
link bandwidth). For TCP, other potential sources of measurement 
noise include TCP segmentation offloading (TSO) and the use of 
delayed ACKs [Hay10]. A congested reverse path may also result in 
an erroneous assessment of the congestion state of the forward 
path. Finally, in the case of fast or short-distance links, the 
majority of the measured delay can in fact be due to processing in 
the involved hosts; typically, this processing delay is not of 
interest, and it can underlie fluctuations that are not related to 
the network at all. 


o Level of statistical multiplexing and RIT sampling: it may be easy 
for an individual flow to "miss" loss/queue overflow events, 
especially if the number of flows sharing a bottleneck buffer is 
significant. This is nicely illustrated, e.g., in Figure 1 of 
[McC08]. 


o Impact of wireless links: several mechanisms that are typical of 


wireless links, like link-layer scheduling and error recovery, may 
induce strong delay fluctuations over short timescales [Gur04]. 
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Interestingly, the results of Bhandarkar et al. [Bha07] seem to paint 
a slightly different picture, regarding the accuracy of delay-based 
congestion prediction. Bhandarkar et al. claim that it is possible 
to significantly improve prediction accuracy by adopting some simple 
techniques (smoothing of RIT samples, increasing the RTT sampling 
frequency). Nonetheless, they acknowledge that even with such 
techniques, it is not possible to eradicate detection errors. Their 
proposed delay-based congestion-avoidance method, PERT (Probabilistic 
Early Response TCP), mitigates the impact of residual detection 
errors by means of a probabilistic response mechanism to congestion- 
detection events. 


2.2. Potential Issues with Delay-Based Congestion Control for LBE 
Transport 


Whether a delay-based protocol behaves in its intended manner (e.g., 
it is "more than TCP friendly", or it grabs available bandwidth ina 
very aggressive manner) may depend on the accuracy issues listed in 
Section 2.1. Moreover, protocols like Vegas need to keep an estimate 
of the minimum ("base") delay; this makes such protocols highly 
sensitive to eventual changes in the end-to-end route during the 
lifetime of the flow [Mo99]. 


Regarding the issue of false positives or false negatives with a 
delay-based congestion detector, most studies focus on the loss of 
throughput coming from the erroneous detection of queue build-up and 
of alleviation of congestion. Arguably, for an LBE transport 
protocol it’s better to err on the "more-than-TCP-friendly side", 
that is, to always yield to _perceived_ congestion whether it is 
"real" or not; however, failure to detect congestion (due to one of 
the above accuracy problems) would result in behavior that is not 
LBE. For instance, consider the case in which the bottleneck buffer 
is small, so that the contribution of queueing delay at the 
bottleneck to the global end-to-end delay is small. In such a case, 
a flow using a delay-based mechanism might end up consuming a good 
deal of bandwidth with respect to a competing standard TCP flow, 
unless it also incorporates a suitable reaction to loss. 


A delay-based mechanism may also suffer from the so-called "latecomer 
advantage" (or "latecomer unfairness") problem. Consider the case in 
which the bottleneck link is already (very) congested. In such a 
scenario, delay variations may be quite small; hence, it may be very 
difficult to tell an empty queue from a heavily-loaded queue, in 
terms of delay fluctuation. Therefore, a newly-arriving delay—based 
flow may start sending faster when there is already heavy congestion, 
eventually driving away loss-based flows [Sha05] [Car10]. 
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3- 


Non-Delay-Based Transport Protocols 


There exist a few transport-layer proposals that achieve an LBE 
service without relying on delay as an indicator of congestion. In 
the algorithms discussed below, the loss rate of the flow determines, 
either implicitly or explicitly, the sending rate (which is adapted 
so as to obtain a lower share of the available bandwidth than 
standard TCP); such mechanisms likely cause more queuing delay and 
react to congestion more slowly than delay-based ones. 


4CP [Liu07], which stands for "Competitive and Considerate Congestion 
Control", is a protocol that provides an LBE service by changing the 
window control rules of standard TCP. A "virtual window" is 
maintained that, during a so-called "bad congestion phase", is 
reduced to less than a predefined minimum value of the actual 
congestion window. The congestion window is only increased again 
once the virtual window exceeds this minimum, and in this way the 
virtual window controls the duration during which the sender 
transmits with a fixed minimum rate. Whether the congestion state is 
"bad" or "good" depends on whether the loss event rate is above or 
below a threshold (or target) value. The 4CP congestion-avoidance 
algorithm allows for setting a target average window and avoids 
starvation of "background" flows while bounding the impact on 
"foreground" flows. Its performance was evaluated in ns-2 
simulations and in real-life experiments with a kernel-level 
implementation in Microsoft Windows Vista. 


The MulTFRC [Dam09] protocol is an extension of TCP-Friendly Rate 
Control (TFRC) [RFC5348] for multiple flows. MulTFRC takes the main 
idea of MulTCP [Cro98] and similar proposals (e.g., [Hac04], [Hac08], 
[Kuo08]) a step further. A single MulTCP flow tries to emulate (and 
be as friendly as) a number N > 1 of parallel TCP flows. By 
supporting values of N between 0 and 1, MulTFRC can be used as a 
mechanism for an LBE service. Since it does not react to delay like 
the protocols described in Section 2 but adjusts its rate like TFRC, 
MulTFRC can probably be expected to be more aggressive than 
mechanisms such as TCP Nice or TCP-LP. This also means that MulTFRC 
is less likely to be prone to starvation, as its aggressiveness is 
tunable at a fine granularity, even when N is between 0 and 1. 


Upper-Layer Approaches 


The proposals described in this section do not require modifying 
transport-—protocol standards. Most of them can be regarded as 
running "on top" of an existing transport, even though they may be 
implemented either at the application layer (i.e., in user-level 
processes), or in the kernel of the end-hosts’ operating systems. 
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Such “upper-layer" mechanisms may arguably be easier to deploy than 
transport-layer approaches, since they do not require any changes to 
the transport itself. 


A simplistic, application-level approach to a background transport 
service may consist in scheduling automated transfers at times when 
the network is lightly loaded, e.g., as described in [Dyk02] for 
cooperative proxy caching. An issue with such a technique is that it 
may not necessarily be applicable to applications like peer-to-peer 
file transfer, since the notion of an "off-peak hour" is not 
meaningful when end-hosts may be located anywhere in the world. 


The so-called Background Intelligent Transfer Service [BITS] is 
implemented in several versions of Microsoft Windows. BITS uses a 
system of application-layer priority levels for file-transfer jobs, 
together with monitoring of bandwidth usage of the network interface 
(or, in more recent versions, of the network gateway connected to the 
end-host), so that low-priority transfers at a given end-host give 
way to both high-priority (foreground) transfers and traffic from 
interactive applications at the same host. 


A different approach is taken in [Egg05] -- here, the priority of a 
flow is reduced via a generic idletime scheduling strategy ina 
host’s operating system. While results presented in this paper show 
that the new scheduler can effectively shield regular tasks from low- 
priority ones (e.g., TCP from greedy UDP) with only a minor 
performance impact, it is an underlying assumption that all involved 
end-hosts would use the idletime scheduler. In other words, it is 
not the focus of this work to protect a standard TCP flow that 
originates from any host where the presented scheduling scheme may 
not be implemented. 


4.1. Receiver-Oriented, Flow-Control-Based Approaches 


Some proposals for achieving an LBE behavior work by exploiting 
existing transport-layer features -- typically, at the "receiving" 
Side. In particular, TCP’s built-in flow control can be used as a 
means to achieve a low-priority transport service. 


The mechanism described in [Spr00] is an example of the above 
technique. Such mechanism controls the bandwidth by letting the 
receiver intelligently manipulate the receiver window of standard 
TCP. This is possible because the authors assume a client-server 
setting where the receiver’s access link is typically the bottleneck. 
The scheme incorporates a delay-based calculation of the expected 
queue length at the bottleneck, which is quite similar to the 
calculation in the above delay-based protocols, e.g., TCP Vegas. 
Using a Linux implementation, where TCP flows are classified 
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according to their application’s needs, Spring et al. show in [Spr00] 
that a significant improvement in packet latency can be attained over 
an unmodified system, while maintaining good link utilization. 


A similar method is employed by Mehra et al. [Meh03], where both the 
advertised receiver window and the delay in sending ACK messages are 
dynamically adapted to attain a given rate. As in [Spr00], Mehra et 
al. assume that the bottleneck is located at the receiver’s access 
link. However, the latter also propose a bandwidth-sharing system, 
allowing control of the bandwidth allocated to different flows, as 
well as allotment of a minimum rate to some flows. 


Receiver window tuning is also done in [Key04], where choosing the 
right value for the window is phrased as an optimization problem. On 
this basis, two algorithms are presented, binary search (which is 
faster than the other one at achieving a good operation point but 
fluctuates) and stochastic optimization (which does not fluctuate but 
converges slower than binary search). These algorithms merely use 
the previous receiver window and the amount of data received during 
the previous control interval as input. According to [Key04], the 
encouraging simulation results suggest that such an application-level 
mechanism can work almost as well as a transport-layer scheme like 
TCP-LP. 


Another way of dealing with non-interactive flows, like web 
prefetching, is to rate-limit the transfer of such bursty traffic 
[Cro98b]. Note that one of the techniques used in [Cro98b] is, 
precisely, to have the downloading application adapt the TCP receiver 
window, so as to reduce the data rate to the minimum needed (thus 
disturbing other flows as little as possible while respecting a 
deadline for the transfer of the data). 


5. Network-Assisted Approaches 


Network-layer mechanisms, like active queue management (AQM) and 
packet scheduling in routers, can be exploited by a transport 
protocol for achieving an LBE service. Such approaches may result in 
improved protection of non-LBE flows (e.g., when scheduling is used); 
besides, approaches using an explicit, AQM-based congestion signaling 
may arguably be more robust than, say, delay-based transports for 
detecting impending congestion. However, an obvious drawback of any 
network-assisted approach is that, in principle, they need 
modifications in both end-hosts and intermediate network nodes. 


Harp [Kok04] realizes an LBE service by dissipating background 
traffic to less-utilized paths of the network, based on multipath 
routing and multipath congestion control. This is achieved without 
changing all routers, by using edge nodes as relays. According to 
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the authors, these edge nodes should be gateways of organizations in 
order to align their scheme with usage incentives, but the technical 
solution would also work if Harp was only deployed in end-hosts. It 
detects impending congestion by looking at delay, similar to TCP Nice 
[Ven02], and manages to improve the utilization and fairness of TCP 
over pure single-path solutions without requiring any changes to the 
TCP itself. 


Another technique is that used by protocols like Network-Friendly TCP 
(NF-TCP) [Arul0], where a bandwidth-estimation module integrated into 
the transport protocol allows to rapidly take advantage of free 
capacity. NF-TCP combines this with an early congestion detection 
based on Explicit Congestion Notification (ECN) [RFC3168] and RED 
[RFC2309]; when congestion starts building up, appropriate tuning of 
a RED queue allows to mark low-priority (i.e., NF-TICP) packets with a 
much higher probability than high-priority (i.e., standard TCP) 
packets, so low-priority flows yield up bandwidth before standard TCP 
flows. NF-TCP could be implemented by adapting the congestion 
control behavior of TCP without requiring to change the protocol on 
the wire -- with the only exception that NF-TCP-capable routers must 
be able to somehow distinguish NF-TCP traffic from other TCP traffic. 


In [Ven08], Venkataraman et al. propose a transport-layer approach to 
leverage an existing, network-layer LBE service based on priority 
queueing. Their transport protocol, which they call PLT (Priority- 
Layer Transport), splits a layer-4 connection into two flows, a high- 
priority one and a low-priority one. The high-priority flow is sent 
over the higher-priority queueing class (in principle, offering a 
best-effort service) using an AIMD, TCP-like congestion control 
mechanism. The low-priority flow, which is mapped to the LBE class, 
uses a non TCP-friendly congestion control algorithm. The goal of 
PLT is thus to maximize its aggregate throughput by exploiting unused 
capacity in an aggressive way, while protecting standard TCP flows 
carried by the best-effort class. Similar in spirit, [Ott03] 
proposes simple changes to only the AIMD parameters of TCP for use 
over a network-layer LBE service, so that such "filler" traffic may 
aggressively consume unused bandwidth. Note that [Ven08] also 
considers a mechanism for detecting the lack of priority queueing in 
the network, so that the non-TCP friendly flow may be inhibited. The 
PLT receiver monitors the loss rate of both flows; if the high- 
priority flow starts seeing losses while the low-priority one does 
not experience 100% loss, this is taken as an indication of the 
absence of strict priority queueing. 
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6. LEDBAT Considerations 


The previous sections have shown that there is a large amount of work 
on attaining an LBE service, and that it is quite heterogeneous in 
nature. The algorithm developed by the LEDBAT working group [Shal11] 
can be classified as a delay-based mechanism; as such, it is similar 
in spirit to the protocols presented in Section 2. It is, however, 
not a protocol -- how it is actually applied to the Internet, i.e., 
how to use existing or even new transport protocols together with the 
LEDBAT algorithm, is not defined by the LEDBAT working group. As it 
heavily relies on delay, the discussion in Sections 2.1 and 2.2 
applies to it. The performance of LEDBAT has been analyzed in 
comparison with some of the other work presented here in several 
articles, e.g. [Aru10], [Car10], [Sch10], but these analyses have to 
be examined with care: at the time of writing, LEDBAT was still a 
moving target. 
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8. Security Considerations 
This document introduces no new security considerations. 
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